Electronic device and audio output method

ABSTRACT

According to one embodiment, an electronic device including, a display, an audio output module, a transmission module, a first detection module, a second detection module, a third detection module, and a controller configured to control at least one of the timing of the transmission of the audio signal by the transmission module and the timing of the output of the first reproduction output by the audio output module in accordance with the time difference detected by the third detection module, and to switch whether or not to control the timing in accordance with the positional relationship between the electronic device and the partner device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromprior Japanese Patent Application No. 2011-263056, filed Nov. 30, 2011,the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an electronic deviceand an audio output method for reproducing content comprising imagessuch as moving images or still images and sound (audio).

BACKGROUND

There has been known a method of enjoying content (which may otherwisebe referred to as a program or a title) comprising images such as movingimages or still images and sound (audio). According to this method,video data being reproduced/displayed by an electronic device (which mayotherwise be referred to as a first apparatus or a source device) istransmitted to a portable terminal device (which may otherwise bereferred to as a second apparatus or a sink device) carried by a user,and reproduced in the portable terminal device.

Meanwhile, according to such a technique, there may be a time differencebetween video and audio output timing in the electronic device (sourcedevice) and video and audio output timing in the portable terminaldevice (sink device).

However, if a delay period is corrected, processing becomes complex dueto the correction of the delay between the sink device and the sourcedevice, and the devices may lose their quick response performance.Therefore, it is preferable that audio reproduction output can besuitably controlled in the sink device and the source device to keep abalance between delay measures and the quick response performance.

BRIEF DESCRIPTION OF THE DRAWINGS

A general architecture that implements the various features of theembodiments will now be described with reference to the drawings. Thedrawings and the associated descriptions are provided to illustrate theembodiments and not to limit the scope of the invention.

FIG. 1A is an exemplary diagram showing an example of radiocommunication connection between a reproducer (electronic device) and areproducer (electronic device) according to an embodiment;

FIG. 1B is an exemplary diagram showing an example of radiocommunication connection between a reproducer (electronic device) and areproducer (electronic device) according to an embodiment;

FIG. 2A is an exemplary diagram showing an example of primary componentsof a first reproducer (source device) and a second reproducer (sinkdevice) according to an embodiment;

FIG. 2B is an exemplary diagram showing an example of the primarycomponents of the first reproducer (source device) and the secondreproducer (sink device) according to an embodiment;

FIG. 3 is an exemplary diagram showing an example of the detection ofthe distance between the reproducers and the setting of audio outputtransmission timing according to an embodiment;

FIG. 4 is an exemplary diagram showing an example of the detection ofthe distance between the reproducers and the setting of audio outputtransmission timing according to an embodiment;

FIG. 5 is an exemplary diagram showing an example of the principle ofthe detection of the distance between the reproducers according to anembodiment;

FIG. 6 is an exemplary diagram showing an example of the principle ofthe detection of the distance between the reproducers according to anembodiment;

FIG. 7 is an exemplary diagram showing an example of the detection ofaudio reproduction outputs (noise determination) of the reproducersaccording to an embodiment; and

FIG. 8 is an exemplary diagram showing an example of the detection(identity) of audio reproduction outputs of the reproducers according toan embodiment.

DETAILED DESCRIPTION

Various embodiments will be described hereinafter with reference to theaccompanying drawings. In general, according to one embodiment, anelectronic device including: a display configured to display images; anaudio output module configured to reproduce an audio signal to output afirst reproduction output; a transmission module configured to transmitthe audio signal to a partner device; a first detection moduleconfigured to detect a second reproduction output reproduced by thepartner device from the audio signal transmitted by the transmissionmodule or to detect an audio output that is output by the partnerdevice; a second detection module configured to detect a parameterregarding the positional relationship between the partner device and theelectronic device; a third detection module configured to detect a timedifference between the second reproduction output of the partner devicedetected by the first detection module and the first reproduction outputthat is output by the audio output module; and a controller configuredto control at least one of the timing of the transmission of the audiosignal by the transmission module and the timing of the output of thefirst reproduction output by the audio output module in accordance withthe time difference detected by the third detection module, and toswitch whether or not to control the timing in accordance with thepositional relationship between the electronic device and the partnerdevice.

Embodiments will now be described hereinafter in detail with referenceto the accompanying drawings.

FIG. 1A and FIG. 1B show an example of connection on a networkcomprising recorder and/or reproducer according to the embodiment.Various elements (components) described below may be provided byhardware or may be provided by software using, for example, a CPU(microcomputer).

In FIG. 1A, a first reproducer (which may otherwise be referred to as asource device) 100 can use a radio communication unit (radiocommunication module/means 1) 101 to pass control signals to a radiocommunication unit (radio communication module/means 2) 201 of a secondreproducer (which may otherwise be referred to as a sink device) 200 andto transmit data (content, images, and sound [audio]) to the secondreproducer 200. The first reproducer 100 and the radio communicationunit (radio communication module/means 1) 101 connected thereto or thesecond reproducer 200 and the radio communication unit (radiocommunication module/means 2) 201 connected thereto may be prepared asone, or may be connected to each other by a predetermined interface.

As long as the first reproducer 100 can display images in accordancewith video signals and sound (audio) signals and can reproduce sound(audio) in accordance with audio signals, the first reproducer 100 mayhave any internal configuration and may be any combination of units ormodules. For example, the first reproducer 100 may be a televisionbroadcast receiver (which may otherwise be simply referred to as atelevision set), a combination of a recorder and a monitor, or acombination of a set top box (STB, recognized as an external tuner) anda monitor.

As long as the second reproducer 200 can display images in accordancewith video signals and image display in accordance with sound (audio)signals and can reproduce sound (audio) in accordance with audiosignals, the second reproducer 200 may have any internal configurationand may be any combination of units or modules. For example, the secondreproducer 200 may be an audiovisual (AV) amplifier (which may otherwisebe referred to as a repeater) and a connected speaker (audioreproducer), an audio device which is a combination of an AV amplifierand a speaker, a combination of a recorder and a monitor, a portableterminal device (for example, a tablet computer, a notebook computer, acellular telephone capable of reproducing video data, and a smartphone),or a television set.

As shown in FIG. 1B, the first reproducer 100 and the second reproducer200 may be connected to each other (capable of exchanging controlsignals and data) via a router (network manager) 1 comprising, forexample, a domestic local area network (LAN). In FIG. 1B, the router 1may enable the (whole) domestic LAN to be a wireless system or therouter 1 may be wireless on part of its connection (path). Nowadays,speed is significantly increased (communication capacity is increased)in a network comprising the router 1. It is possible to use near-fieldradio communication standards such as Bluetooth (registered trademark)and power line communication (PLC). The router 1 may function as aserver.

Primary components of the first reproducer (source device) and thesecond reproducer (sink device) shown in FIG. 1A are illustrated in FIG.2A.

The source device (television set) 100 comprises a main controller (mainprocessing unit [MPU] or main control block), or control module 1) 111,a recorder (recording module 1) 112, a signal processing module (digitalsignal processor [DSP] 1) 113, a display unit (monitor module (videodisplay device or display) 1) 114, a source-sink positional relationshipdetection module (positional relationship detection module 1) 115, adelay period detection control circuit (timestamp detection controlcircuit, delay detection module 1) 116, a speaker (audio reproductionmodule 1) 117, a microphone (acoustic signal detection module 1) 118,and an external input module (input module 1) 119 for receiving theinput of video signals and sound (audio) signals, that is, content. Themain controller (control module 1) 111 is connected to the radiocommunication unit (radio communication module/means 1) 101. The radiocommunication unit (radio communication module/means 1) 101 does notexclusively communicate directly with the communication partner device(second reproducer 200), and has only to be able to communicate with therouter 1, as in the example shown in FIG. 1B.

The main controller (control module 1) 111 comprises, for example, arecording/reproduction processing module (management informationprocessing module/timestamp processing module), an encode parameterdetection/processing module, a graphical user interface (GUI) displaycontrol module, and an SDRAM (main memory). The main controller (controlmodule 1) 111 controls the elements in the reproducer 100, handles useroperations (input instructions) through, for example, a remotecontroller, and controls the recording, reproduction, copying, moving,erasing, and editing of content (programs). The main controller (controlmodule 1) 111 also controls the communication between the radiocommunication unit 101 and the communication partner device. The radiocommunication unit 101 can independently handle the reception of awake-up command coming from the communication partner device and awake-on-LAN command coming from the communication partner device via therouter 1, and also handle part of the processing compliant with thereceived command. In accordance with the result of the processing, theradio communication unit 101 supplies a predetermined control signal toa specific component of the source device 100 through the maincontroller 111.

For example, the recorder (recording module 1) 112 records, in apredetermined format, content which has been processed into apredetermined format in the DSP 113 or the main controller 111 and whichhas been input through the input module 119, and programs obtained bydemodulating broadcast waves. The recorder 112 holds, as theabove-mentioned information, various files in the form of digitalsignals. As the files, various kinds of data, for example, audio/videocontent, or text data such as characters and picture data are widelyknown. Regarding video (moving images), for example, Moving PictureExperts Group (MPEG) 1, 2, and 4 (H.264/AVC) are widely known. Regardingaudio (sound/music), for example, MPEG audio layer 3 (MP3), audio codenumber 3 (AC3), and linear pulse-code module (PCM) are widely known.Regarding still images, for example, Joint Photographic Experts group(JPEG) is widely known. Nowadays, a semiconductor memory (solid-statedrive [SSD] such as a Secure Digital [SD] card) can be used as arecording medium.

Mainly under the control of the controller 111, the DSP 113 uses theSDRAM as a work memory, for example, to transfer data (content), createthumbnail images (reduced images) corresponding to given images (stillimages or moving images) of content, and generate video signals that canbe displayed by the display unit 114 and audio signals that can bereproduced by the speaker 117.

The display unit 114 reproduces the video signals supplied from the DSP113, and displays images visible to the user. The display on the displayunit 114 may be display in the form of GUI images output by the GUIdisplay control unit that are superposed on one another withpredetermined transmissivity (for example, to wait for an instructioninput by the user).

Although described later in detail, the source-sink positionalrelationship detection module (positional relationship detection module1) 115 estimates (detects) the distance from the communication partner(for example, the second reproducer 200) with which the radiocommunication unit 101 communicates, and the presence of a boundary (forexample, a wall in a house) therebetween.

The delay period detection control circuit (timestamp detection controlcircuit, delay detection module 1) 116 controls whether to execute delaycorrecting processing in accordance with the distance estimated by oneof the source-sink positional relationship detection module (positionalrelationship detection module 1) 115 or the source-sink positionalrelationship detection module (positional relationship detection module2) 215 and the presence of a boundary. The delay period detectioncontrol circuit 116 also detects a delay period relative to videodisplay and audio output in its device (source device 100) when thesource device 100 transmits content data to the second reproducer 200for video display and audio output. The delay period detection controlcircuit 116 then stores the delay period in a memory. In detecting thedelay period, the delay period detection control circuit 116 detects atime difference (delay period, delay time) between sound output by thesink device 200 and sound output by the device 100.

For example, the delay period is detected as follows. First, whencontent data which is not delay-corrected is read, the device 100decodes and outputs the content data, and transmits the content data tothe sink device 200 to output the sound of the content data. Here, thesink device 200 decodes and outputs the content data on receipt of thecontent data, so that the source device 100 detects the sound output bythe sink device 200, and detects a time difference between the soundoutput by the sink device 200 and the sound output by the source device100. The content data used for the detection of the delay period may be,for example, content data including a model sound for delay perioddetection, or content data for normal content (a movie or broadcastcontent).

When correcting a delay, the delay period detection control circuit 116controls the timing of video display and audio output in the device 100in accordance with the delay period stored in the memory. That is, whenthe display/output timing in the sink device 200 is delayed relative tothe display/output timing in the device (source device 100), the sinkdevice 200 delays its video display and audio output in conformity tothe display/output timing in the source device 100. On the other hand,when the timing of video display and audio output in the sink device 200is earlier than the timing of video display and audio output in thedevice 100, the radio communication unit 101, for example, is used toreport the delay period to the sink device 200 and instructs the sinkdevice 200 to delay its video display and audio output by the delayperiod. The delay period here is the delay period of the output timingin the source device 100 relative to the output timing in the sinkdevice 200.

The delay period detection control circuit 116 may detect a delay periodfor each encoding format of content data, and store the detected delayperiod for each encoding format in the memory. This is attributed to thefact that the time required for the sink device 200 to receive contentdata and output images and sound may vary depending on the encodingformat. The time required for the source device to read content data andoutput images and sound may vary depending on the encoding format.

In this case, when the content data is transmitted to the sink device,the delay period detection control circuit 116 may identify the encodingformat of the content and correct the delay period in accordance withthe identified encoding format. The delay period detection controlcircuit 116 may correct even contents in the same encoding format bydifferent delay periods depending on whether the contents arecopyright-protected contents. The protected content is encoded by apredetermined encoding format and transmitted. Therefore, when theprotected content is processed, encoding processing in the source device100 and decoding processing in the sink device may be added to theprocessing of unprotected content. That is, the delay period between theoutput timing in the source device 100 and the output timing in the sinkdevice 200 may vary depending on whether content is protected.Therefore, the delay period detection control circuit 116 may detectdelay periods for the protected content and the unprotected contentregarding content data in the same encoding format, and store the delayperiods. When the content is transmitted, the delay period detectioncontrol circuit 116 may perform suitable delay period control dependingon the encoding format and depending on whether the content isprotected.

The speaker (audio reproduction module 1) 117 reproduces sound (audio)from the audio signal supplied from the DSP 113, and outputs sound(audio) audible to the user.

The microphone (acoustic signal detection module 1) 118 detects, forexample, the speech of the user and sound (audio) output by thecommunication partner device, and is used to generate sound (audio) dataavailable to the source-sink positional relationship detection module115 to estimate (detect) the distance from the communication partner(for example, the second reproducer 200) and the presence of a boundary(for example, a wall in a house), or used for one user instructioninputting method called voice input.

In the source device 100, the source-sink positional relationshipdetection module 115, the delay period detection control circuit 116,and the microphone 118 can be omitted depending on, for example, the useand size (stationary/portable) of the reproducer. For example, themicrophone 118 can double as the speaker 117.

The primary components of the first reproducer (source device) and thesecond reproducer (sink device) shown in FIG. 1B are illustrated in FIG.2B.

The sink device (portable terminal device) 200 comprises a maincontroller (control module 2) 211, a recorder (recording module 2) 212,a signal processing module (digital signal processor [DSP] 2) 213, adisplay unit (monitor module (video display device or display) 2) 214, asource-sink positional relationship detection module (positionalrelationship detection module 2) 215, a delay period detection controlcircuit (timestamp detection control circuit, delay detection module 2)216, a speaker (audio reproduction module 2) 217, a microphone (acousticsignal detection module 2) 218, and an external input module (inputmodule 2) 219 for receiving the input of video signals and sound (audio)signals, that is, content. The main controller (control module 2) 211 isconnected to the radio communication unit (radio communicationmodule/means 2) 201. The radio communication unit (radio communicationmodule/means 2) 201 does not exclusively communicate directly with thecommunication partner device (first reproducer 100), and has only to beable to communicate with the router 1, as in the example shown in FIG.1B.

The configurations and operations of the main components of the sinkdevice 200 shown in FIG. 2B are similar to those of the source deviceillustrated in FIG. 2A (components having the same names function in asubstantially similar manner) and are therefore not described.

Although described later in detail, the source-sink positionalrelationship detection module (positional relationship detection module2) 215 estimates (detects) the distance from the communication partner(for example, the first reproducer 100) with which the radiocommunication unit 201 communicates, and the presence of a boundary (forexample, a wall in a house) therebetween.

Substantially as in the example that has already been described withreference to FIG. 2A, the delay period detection control circuit(timestamp detection control circuit, delay detection module 2) 216controls whether to execute delay correcting processing in accordancewith the distance estimated by one of the source-sink positionalrelationship detection module (positional relationship detection module2) 215 or the source-sink positional relationship detection module(positional relationship detection module 1) 115 and the presence of aboundary. The delay period detection control circuit 216 also detects adelay period of video display and audio output in the sink devicerelative to video display and audio output in the source device whenimages are displayed and sound is output on receipt of content data fromthe source device. The delay period detection control circuit 216 thenstores the delay period in a memory. In detecting the delay period, thedelay period detection control circuit 216 detects a time difference(delay period, delay time) between sound output by the source device andsound output by the sink device.

For example, the delay period is detected as follows. First, whencontent data which is not delay-corrected is received from the sourcedevice 100, the sink device 200 decodes and outputs the content data.Here, the source device 100 also decodes and outputs the content data,so that the sink device 200 detects the sound output by the sourcedevice 100, and detects a time difference between the sound output bythe source device 100 and the sound output by the sink device 200. Thecontent data used for the detection of the delay period may be, forexample, content data including a model sound for the detection of thedelay period, or may be normal content.

When correcting a delay, the delay period detection control circuit 216controls the timing of video display and audio output in the device 200in accordance with the delay period stored in the memory. That is, whenthe output timing in the sink device 200 is delayed relative to theoutput timing in the source device 100, the sink device 200 reports thedelay period to the source device 100 and instructs the source device100 to delay its video display and audio output by the delay period. Onthe other hand, when the timing of video display and audio output in thesink device 200 is earlier than the timing of video display and audiooutput in the source device 100, the sink device 200 delays its videodisplay and audio output by the delay period, or reports the delayperiod to the source device 100 to delay the transmission timing of thecontent data by the delay period.

The delay period detection control circuit 216 may detect a delay periodfor each encoding format of content data, and store the detected delayperiod for each encoding format in the memory. In this case, when thecontent data is transmitted to the sink device, the delay perioddetection control circuit 216 may identify the encoding format of thecontent and correct the delay period in accordance with the identifiedencoding format. The delay period detection control circuit 216 maycorrect even contents in the same encoding format by different delayperiods depending on whether the contents are copyright-protectedcontents. That is, the delay period detection control circuit 216 maydetect delay periods for the protected content and the unprotectedcontent regarding content data in the same encoding format and store thedelay periods. When images are displayed and sound is output on receiptof content, the delay period detection control circuit 216 may performsuitable delay period control depending on the encoding format anddepending on whether the content is protected.

In the sink device 200, the display unit 214, the source-sink positionalrelationship detection module 215, the delay period detection controlcircuit 216, and the microphone 218 can be omitted depending on, forexample, the use and size (stationary/portable) of the reproducer. Forexample, the microphone 218 can double as the speaker 217.

The primary operation according to the embodiment is shown in FIG. 3 ina software fashion.

In the interconnection between the source device and the sink deviceshown in FIG. 1A and FIG. 2A or in FIG. 1B and FIG. 2B, various outputforms are produced regarding audio reproduction (audio output) in thesink device. For example, the sink device reproduces sound independentlyof the source device, or the sink device does not reproduce sound (audiooutput is set to “0”) for the following reasons:

(A) the source device and the sink device are located in the same room(space);

a) sound output by the source device is sufficiently audible in theposition of the sink device;

b) sound output by the source device is sufficiently audible in theposition of the sink device, and subtitle indication, for example, isreproduced in the sink device;

c) sound output by the source device is sufficiently audible in theposition of the sink device, and voices in different languages arereproduced in the sink device;

d) the sink device is too far from the source device to hear soundoutput by the source device, and cannot hear the sound output by thesource device (in the position of the sink device);

e) there is a barrier (for example, an ornamental plant which does notblock sound) between the sink device and the source device; and

f) there is a barrier (for example, a bookshelf which blocks sound)between the sink device and the source device, and

(B) the source device and the sink device are located in different rooms(spaces);

a′) sound output by the source device is sufficiently audible in theposition of the sink device;

c′) sound output by the source device is sufficiently audible in theposition of the sink device, and voices in different languages arereproduced in the sink device; and

d′) sound output by the source device is not audible in the position ofthe sink device.

When the sink device is a portable device, for example, there is also acase where

C) the sink device is driven by a non-AC (commercial) power source suchas a battery or a fuel battery.

That is, as shown in FIG. 3, the sink device is detected to be locatedin a delay correction area [311],

the delay correction is detected to be necessary when the sink device isdetected to be located in the delay correction area [312],

the delay correction is detected to be executable [313], and

a “delay correction” is made to the sound reproduced by the sink device[314],

so that the audio reproduction (audio output) in the sink device can beset to a suitable form (lip-sync with reproduced images).

As described above, the “delay correction” is often made to eliminatethe difference between the sound transmitted by the source device andreproduced by the sink device and the sound reproduced by the sourcedevice. For example, the “delay processing” may be performed for thevideo reproduction by the sink device depending on, for example, thespeed of video decoding (time required for decoding) by the sink device.There is a possibility that the “delay correction” is substantiallyunnecessary depending on the speed of audio decoding (time required fordecoding) by the sink device.

On the other hand, when the sink device is detected to be located in thedelay correction area [311—YES] and when the delay correction isdetected to be not necessary (the “delay correction” is unnecessary)[312—NO], current reproduction setting (conditions) is maintained.

When the sink device is detected to be located in the delay correctionarea [311—YES] and when the delay correction is detected to be necessary[312—YES], it is detected that, for example, “delay reductionprocessing” for stopping the audio output is executable [315] if thedelay correction is detected to be not executable [313—NO].

When the “delay reduction processing” is detected to be executable[315—YES], a device to preferentially stop audio output, for example, abattery-driven (non-AC-driven) device is detected [316], and the audiooutput of this device is preferentially stopped [317]. When there is nodevice to preferentially stop audio output [315—NO], the audio outputfrom one of the sink device and the source device that allows its audiooutput to be stopped (the device which has no problem or which makeslittle difference if its audio output is stopped) is stopped [318].

When the sink device is not located in the delay correction area[311—NO], it is detected that the “delay correction” is executed inconsideration of the case where the “delay correction” is alreadyexecuted in the target sink device [319].

That is, when the device which has been located in the delay correctionarea moves out of this area, for example, because of the movement of theuser, the already executed “delay correction” is unnecessary. Therefore,when the position of the sink device is detected to be out of the areatargeted for the “delay correction”, it is preferable to finish the“delay measures (delay correction)” (to output sound by thesynchronization of the source and the sink) [320].

The “delay measures (delay correction)” described above with referenceto FIG. 3 can be performed by the sink device and the source device thatpermit the “delay measures (delay correction)”. Therefore, regardinginitial setting [301], it is preferable to previously check, forexample, the following:

1) a delay correcting circuit is prepared (installed);

2) the source device can be synchronized with the sink device;

3) a period X of ascertaining whether a delay correction is made can beindependently set (X is a numerical value prescribed by a predeterminedmodule), and

4) information on the capability of the device.

Moreover, as described above, the sink device may not be adapted to the“delay correction”, but may accept “correction reduction measures” thattypically comprise the stopping of the audio output. Therefore, in theinitial information (initial setting), it is preferable to acquire

5) information on whether the partner device is a device adapted toenergy saving, that is, a non-AC-driven (battery-driven) device.

When it is found out from the initial information (initial setting) thatthe “delay correction” is executable, it is preferable that a“predetermined waiting time” corresponding to a checking time necessaryto determine whether to make a correction is reserved (the transmissionof content is suspended for a given length of time) at the time of thetransmission of content from the source device to the sink device.

Furthermore, it is preferable that the above-mentioned informationregarding the execution (or cancellation) of the “delay correction” isheld in a recording medium (for example, a firmware in the maincontroller, or an SDRAM). Considering a small number of factors thatchange the source-sink distance, for example, in the home, theabove-mentioned information regarding the execution of the “delaycorrection” may be recorded in an unshown nonvolatile memory used, forexample, to hold timer recording information.

The sequence for implementing the embodiment shown in FIG. 3 isdescribed in FIG. 4.

First, the source device outputs a “beacon” (detection signal) at everypredetermined period to check the presence of the sink device [401].

When the sink device responds to the “beacon”, the source device and thesink device respectively recognize their partner devices (devicediscovery [detection]) [402].

Furthermore, the source device and the sink device respectively certifythat their partner devices permit a “delay correction”/“delay measures”(authentication/association <agreement, confirmation>) [403].

The devices then mutually check “device capability information” bycapability exchange [404].

The source device then starts “inquiring of the sink device” [405]. Inaccordance with the response from the sink device, the source deviceuses a later-described checking method, for example, using radio orother detection module, to check whether the sink device is located inthe correction area [406].

The sink device also reports, for example, the presence of a currentlyused “delay correction” (delay information exchange) [407].

The source device which has received a report from the sink device as tothe presence of the currently used “delay correction”, and the sinkdevice repeat the “delay correction/delay reduction instruction” and“current delay information exchange”, for example, regarding whether topermit the setting of the use of the “delay correction”, a “correctionamount” when the “delay correction” is used, and whether to execute a“delay reduction measures” when the “delay correction” cannot be used[408-1 to 408-m (m is a positive integer), 409-1 to 409-n (n is apositive integer)].

As a result, the amount of the “delay correction” made to the sinkdevice or the source device or the execution of the “delay reductionmeasures” are set.

As has already been described with reference to FIG. 3, the position ofthe sink device may change with the movement of the user. Therefore, itis preferable that the above-described [405] to [409-n] are repeatedperiodically (at every given period).

When it is detected by repeating the above-described [405] to [409-n]that the position of the sink device has changed and the sink device islocated out of the “delay correction” area, whether the cancellation ofthe “delay correction/delay reduction measures” already described withreference to FIG. 3 is needed is checked. When it can be detected thatthe cancellation is needed (the “delay correction/delay reductionmeasures” is unnecessary) [411], a cancellation instruction is given tothe sink device or the source device to which the “delay correction” ismade [412].

The currently executed “delay correction/delay reduction measures” isfinished in the sink device or the source device to which thecancellation instruction is given. As a result, “cancellationcompletion” is reported to the partner device [413].

Now, an example of the method of detecting whether the sink device islocated in or out of the “delay correction” area is shown.

FIG. 5 shows the relationship between the volume (energy) of soundacquired by the microphone (sound (audio) output detection module) 118of the source device 100 (the microphone 218 of the sink device) and theintensity or electric field intensity (power) of a signal received bythe radio communication unit 101 (201 in the sink device).

As apparent from FIG. 5, the volume of sound is substantially indirectly proportion to the intensity of the radio signal. Thus, forexample, the sound reproduced by the partner device (or a startup melodyoutput when power is applied) is acquired by the microphone mostlyprovided in the source device or the sink device. From the volume of thesound, the distance from the partner device can be estimated.

FIG. 6 shows the relationship between the volume of the sound (startupmelody) of the partner device detected by the microphone (sound (audio)output detection module) and the inter-device distance.

In a special case where there is a barrier (for example, a bookshelfwhich blocks sound) between the devices, the damping level (dampingamount) of the sound (startup melody) acquired by the microphone isproportional to the square of the inter-device distance (reaches onepart of the square of the distance as the distance increases) (generallyin Line-Of-Sight (LOS)).

Therefore, if it is known that the volume of the sound (startup melody)output by the sound source (partner device) is under a predeterminedcondition, the inter-device distance can be estimated in accordance withthe volume of the sound (startup melody) detected by the microphone. Onthe other hand, it is necessary that the sound (startup melody) detectedby the microphone be the sound (startup melody) corresponding to thesignal transmitted from the transmitter (it is necessary to considerthat the sound acquired by the microphone is only noise). An“auto-correction function (ACF)” described below with reference to FIG.7 and FIG. 8 is used to specify that the sound (startup melody) acquiredby the microphone is the sound used to measure (detect) the distancefrom the partner device, thereby detecting the inter-device distance.

That is, the relationship between a time function p(t) and ACF can befound by Equation (1) to (3) wherein “t” is a delay time and “2T” is anintegral interval. It is recognized that ACF and the power spectrum havemathematically the same information when Pd(ω) is a power spectrum of asound source signal.

$\begin{matrix}{{\Phi_{\rho}(\tau)} = {\lim\limits_{\tau->\infty}{\frac{1}{2T}{\int_{- \tau}^{+ \tau}{{p(t)}{p\left( {t + \tau} \right)}{t}}}}}} & (1) \\{{\Phi_{\rho}(\tau)} = {\int_{- \infty}^{+ \infty}{{P_{d}(\omega)}^{j\omega\tau}{\omega}}}} & (2) \\{{P_{d}(\omega)} = {\int_{- \infty}^{+ \infty}{{\Phi_{d}(\tau)}^{- {j\omega\tau}}{\tau}}}} & (3)\end{matrix}$

Equation (1), Equation (2), and Equation (3) are general formulas, andare shown at, for example, [www.ymec.com/hp/signal/acf.htm].

On the other hand, as shown in FIG. 7, ACF cuts out the signal p(t)which is as long as the integral interval “2T”, so that a signal onlydelayed in time can be detected as p(t+1). That is, when p(t) and p(t+1)have high amplitude and are similar repetitive components, it can bedetected that their ACFs (correlation value Fp(t)) increase, so thatp(t) and p(t+1) are highly similar. Noise components (noise) differentfrom the sound decrease in the ACF value in a short time as shown by wayof example FIG. 8. Therefore, the similarity to the sound (startupmelody) can be detected by using ACF.

More specifically, a real audio signal obtained from audio (sound) whichis transmitted through the speaker of the source device and which iscollected by the microphone of the sink device is recorded for a givenlength of time. The correlation between the real audio signal and anaudio signal which has arrived in the sink device by radio and decodedtherein is calculated in the predetermined interval 2T. The start timeof the interval 2T is sequentially shifted to calculate a timedifference which is highly correlative, that is, which permit thehighest similarity of waveforms. When the used signal has a change on atime axis, the time difference having the highest similarity ofwaveforms is a delay time, and is a correction amount on the time axis.The correction amount on the time axis, that is, the above-mentionedtime difference corresponds to the difference between the soundtransmitted by radio and the microphone-collected sound that may beactually audible to a person (user). When the user has the sink deviceat hand, sound is collected by the sink device at hand so that a delaytime can be determined in such a manner as to take into considerationthe delay of sound attributed to the source-sink distance. That is, adelay period in a more natural state can be determined so that soundarrives at a distance later. An audio signal necessary for a correctiondoes not necessarily have to be, for example, a periodic pulse fortraining (for the repetition of the operations in [405] to [409-n] whichwill be described later with reference to FIG. 4). If theabove-mentioned ACF is used, a correction amount can be determined byusing actually traveling sound which has a given waveform.

The volume of the sound (startup melody) changes with the reproductionlevel (volume setting) of the partner device and the sensitivity (gain)of the microphone. It is thus preferable to set a correction amount (andthe inter-device distance) by repeating (training) the operations in[405] to [409-n] shown in FIG. 4.

On the other hand, an actual delay period can be easily found bycomparing a timestamp acquired from the content by the delay perioddetection control circuit (timestamp detection control circuit, delaydetection module 1) 116 of the source device 100 (see FIG. 1A, FIG. 1B,and FIG. 2A), that is, timestamp information in a packet identifier PIDattached to a TS packet of a transport stream (TS), with timestampinformation acquired from the content by the delay period detectioncontrol circuit (timestamp detection control circuit, delay detectionmodule 2) 216 of the sink device 200 (see FIG. 1A, FIG. 1B, and FIG.2B).

That is, [405] to [409-n] described above with reference to FIG. 4 arerepeated in the above-mentioned training to estimate a distance, set adelay period that conforms to the estimated distance, and determinewhether the delay period set by the sound (audio) output reproduced bythe partner device is a delay period optimum to the estimated distance.This can prevent the user from having difficulty in hearing the sound,for example, due to a time difference in sound (audio) reproductionoutputs between the source device and the sink device that are locatedat a predetermined distance from each other.

The inter-device distance between the source device and the sink deviceis thus found. As a result, various output forms can be set for theaudio reproduction (audio output) in the sink device (source device) inaccordance with the instruction from the user or the position of theuser. For example, the sink device (or the source device) reproducessound independently of the source device (or the sink device), or thesink device (or the source device) does not reproduce sound (audiooutput is set to “0”), in accordance with the various factors classifiedinto (A)a) to (A)f) and (B)a′), (B)c′), and (B)d′) as described above.

For example, suppose that family members gather. There is a televisionset having a large screen for reproducing content. For example, one ofthe family members has a tablet device (sink device) at a distance fromthe source device. This person views images displayed by the tabletdevice and listens to the sound reproduced by the tablet device. Theperson having the tablet device simultaneously (redundantly) hears, witha time difference, the sound reproduced by the tablet device and thesound which is reproduced by the source device and which directlyapproaches the tablet device. In this case, a “delay correction” is madeto one of the audio outputs. This can prevent the problem of redundantlyhearing the sounds reproduced by the respective devices.

Furthermore, for example, when a person enjoying content by the tabletdevice is located in the vicinity of the source device, the sound outputby the tablet device is set to “0”. Otherwise, for example, when morethan one user use their tablet devices to reproduce the same content,one of the tablet devices only outputs sound (audio). This can inhibittwo or more audio outputs that are redundantly audible with discomfortfor the same reproduced images.

The user having the tablet device (sink device) may move, and theinter-device distance between the tablet device and the source devicemay change. In this case, [405] to [409-n] described above withreference to FIG. 4 are repeated to set a new a “delay correction”amount, or the “delay correction” is canceled. This can inhibit two ormore audio outputs that are redundantly audible with discomfort for thesame reproduced images.

The above-described embodiment can be carried out in various formsdescribed below.

For example, the above-described embodiment can be carried out, forexample, by a source device provided to display images or generatesound, and a sink device provided to receive the images or audio signalinformation by radio. The source device 100 comprises the display unit1, the radio module (1) 101, the control module (1) 111, the recordingmodule (1) 112, the speaker (1) 117, and the external input module (1)119. The sink device 200 comprises a radio module (2) 201, the speaker(2) 217, the control module (2) 211, and the external input module (2)219. One of the source device and the sink device comprises at least onemicrophone, a delay period detection control circuit or timestampdetection control circuit, and source-sink positional relationshipdetection module. Sound generated by the speaker (1) 117 or the speaker(2) 217 attached to one of the source device and the sink device isreceived by the microphone (1) 118 or the microphone (2) 218 attached tothe other device in accordance with information obtained by the delayperiod detection control circuit or timestamp detection control circuit116 or the delay period detection control circuit or timestamp detectioncontrol circuit 216 shown in FIG. 2A and/or FIG. 2B. The delay period ofthe sound or the images on the time axis is determined, and the delay iscontrolled. The source-sink positional relationship detection module 115or the source-sink positional relationship detection module 215 controlswhether to control the delay and controls the on/off of the speaker (1)117 or (2) 217.

That is, the sound generated from one speaker is detected by the othermicrophone to determine a delay period so that sound can be output fromone or both of the microphones. The synchronization of sounds can belinked to the synchronization of images by the lip-sync of sounds andimages in the sink device and the source device. Moreover, thesource-sink positional relationship detection module can turn on/off thedelay control. This advantageously enables automatic control. Forexample, when persons are in the same room, a synchronization mode isset. When persons are in different rooms, a delay is minimized byindependent operations.

Furthermore, the above-described embodiment can be carried out, forexample, by using the source-sink positional relationship detectionmodule as a means for determining whether to control the sound delay andwhether to control the on/off of the speaker (1) 117 or (2) 217.

As a result, the positional relationship between the source and the sinkis known. Therefore, when the source and the sink are close to eachother, the delay is controlled, or sound is only output from onemicrophone. On the other hand, when the sink and the source are locatedapart from each other, the delay control is unnecessary for the user, orthe delay control can rather be turned off to prioritize the quickresponse performance.

Furthermore, the above-described embodiment can be carried out, forexample, if the source-sink positional relationship detection module isposition specifying module that uses an audio microphone, a camera,radio reception sensitivity/reception quality, a motion sensor, a globalpositioning system (GPS) reception output, and radio charge collectionsystem (Paid) network base station information or wireless local areanetwork (LAN) base station information, or is composite positionspecifying module that uses the combination of the above.

That is, when the source device and the sink device are at a distancefrom each other, for example, are in different rooms, any active(available/on-state) module/means can be used for detection withoutusing exclusive detection module.

Furthermore, the above-described embodiment can be carried out, forexample, by preparing detection values of more than one detection moduleas a conversion table.

That is, the advantage is that the positional relationship between thesource device and the sink device can be known by the positionalrelationship detection module which is normally in operation and thesound (audio) can be controlled. For example, it is known that theenergy amount of the sound received by the microphone is determined ininverse proportion to the square of the distance from the sound source.On the other hand, it is known, for example, from a Friis formula thatthe radio signal used to transmit images/sound from the source device tothe sink device is damped in inverse proportion to the square of thedistance from the sound source. This proves that if the energy of thereceived sound is set to a horizontal axis and received radio signalpower is set to a vertical axis, the relationship therebetween can beapproximated by a straight line that passes through the origin in thecase of a clear interval without any blockage. As has been describedwith reference to FIG. 5, whether to correct the delay period can bedetermined by using a predetermined threshold as a standard.

Furthermore, the above-described embodiment can be carried out, forexample, if the delay period detection control circuit/timestampdetection control circuit compares the sound (audio) signal [A]generated in the speaker (1) 117 or the speaker (2) 217 of one of thesource device and the sink device with the sound (audio) signal [B]output from a speaker different from the above speaker, and therebydetermines a delay period to be corrected and determines whether acorrection is needed.

That is, when there is no delay of sound, the correlation valuesrelative to mathematic characteristics (for example, a sliding amountwhen a sliding correlation is calculated) of the audio signal [A]generate from the speaker (1), for example, and the audio signal [B]generate from the speaker (2), for example, behave in a similar manner.On the other hand, when there is a delay of sound, a delay period thatapproaches a condition having a single sound source is estimated byusing the fact that the correlation values do not correspond to eachother (estimated by auto-correlation).

For example, the auto-correlation can be found by Equation (4)

z(τ)=∫_(−∞) ^(+∞) h(t)x(t+τ)dt  (4)

wherein the correlation of two functions h(t) and x(t) is represented byz(τ), and z(τ) is obtained by shifting and multiplying h(t) and x(t) byτ and integrating the result within a predetermined interval.

Furthermore, the above-described embodiment can be carried out, forexample, if the delay period detection control circuit/timestampdetection control circuit is obtained by hardware or softwareprocessing. That is, even when no hardware is provided, theabove-described embodiment can be carried out by a software package.

Furthermore, the above-described embodiment can be carried out, forexample, if speakers are provided in both the source device and the sinkdevice, and the source-sink positional relationship detection module isan audio microphone. When a detection audio output of the speaker of thesource device or the sink device is beyond a first threshold, one of thespeakers is only used. When the detection audio output is equal to orless than a second threshold, both the speakers of the source device andthe sink device are turned on. That is, the problem of sound delays canbe avoided by a simple configuration. For example, when the devices arein the same room, sound is only output from one device. When the devicesare in different rooms and sound does not reach, sounds are output fromboth the devices.

Furthermore, the above-described embodiment can be carried out, forexample, if the output from the speaker of one of the source device andthe sink device is the output from the speaker attached to the devicewhich is not battery-driven when one of the devices is battery-driven.That is, the consumption of the electricity of the battery-driven devicecan be prevented.

Furthermore, the above-described embodiment can be carried out, forexample, if, as a means of measuring a delay period on the time axis, adelay period to be corrected is found by measuring more than one stateas the states of the sound source depending on whether sound (audio) isoutput from the speaker (1) 117 or the speaker (2) 217 and finding theauto-correlations on the time axis. That is, the pitch of sound and thestrength of the pitch of the sound can be known by the auto-correlationson the time axis, and can be compared with those in a condition with onesound source to obtain a delay period to be found. The auto-correlationscan be easily found by sampling and filtering sounds collected by themicrophone, recording the sounds in a register (which may otherwise be asignal processing unit or its firmware) at a sampling pitch or itsintegral multiple in order of time, multiplying the same data atdifferent positions of the register and thereby figuring out the sum ofthe values. In the case of one sound source, the auto-correlation has asingle peak when the shift amount (delay time) of the register is set toa horizontal axis. Even when there are two sound sources, the delayperiod is zero and the peak is at the same position as that in the caseof one sound source if sounds from the two sound sources reach themicrophone at the same time. When two sound sources are delayed, thereare more than one peak. Thus, the waveform of one sound source is usedas a reference value, and a shift amount of the register thatcorresponds to the reference value is found to obtain a delay time to becorrected. Therefore, the source device and the sink device are inlip-sync (the synchronization of images and sound), so that if the soundis synchronized with the sink device, the synchronization of images isimproved accordingly.

Furthermore, the above-described embodiment can be carried out, forexample, by creating the conversion table in accordance with gainedvalues (acquired values) of more than one of the detection module at thesame time. In this case, the conversion table can be easily created fromthe values obtained at the same time, that is, obtained when thepositional relationship between the source device and the sink device isunique. Thus, whether the device is located in the correction area canbe determined.

Furthermore, the above-described embodiment can be carried out, forexample, by using, as the source-sink positional relationship detectionmodule, detection module other than a training period for creating theconversion table and other than audio information. That is, thereception of sound by the microphone is finished at the end of thetraining period, and whether the device has entered the area can beeasily determined, for example, by normally used radio receptionintensity.

Furthermore, the above-described embodiment can be carried out, forexample, by a mechanism and a circuit configuration in which at leastone of the microphones doubles as the speaker (1) or the speaker (2),and the functions of the microphone and the speaker are switched andused on the time axis. That is, one component doubles as the microphoneand the speaker in many mobile information terminals, which can be usedto carry out the above-described embodiment.

Furthermore, the above-described embodiment can be carried out, forexample, by replacing the source-sink positional relationship detectionmodule with positional relationship detection module which is locatedbetween one of the source device and the sink device that greatlychanges in radio signal quality with movement and a radio signalrepeater, when a radio signal is transmitted from the source device tothe sink device via the radio signal repeater. That is, when a radiosignal is sent via an access point (AP), the device that changes in aradio channel between the source and the sink is used as the detectionmodule such that the positional relationship can be more preciselyknown.

Furthermore, the above-described embodiment can be carried out, forexample, by using at least one of the microphones of the source deviceand the sink device that is closer to the user. That is, a more suitablecorrection can be made to a sound delay including reverberations fromthe surroundings by using the position of the microphone close to theuser as a standard position for sound delay correction. For example,when the source device and the sink device are 10 meters apart from eachother, a delay of about 30 milliseconds is generated in the arrival timeof sound if converted with a sound speed of 300 m/second. Therefore, thecorrection amount changes depending on whether to use the microphone ofthe source device or the sink device, so that a highly precise delaycorrection can be made by selecting a proper microphone closer to theuser.

Various embodiments of the systems described herein can be implementedas software applications, hardware and/or software modules, orcomponents on one or more computers, such as servers. While the variousmodules are illustrated separately, they may share some or all of thesame underlying logic or code.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

In the embodiment described above, the source device is, for example, atelevision receiver (TV), but is not limited thereto. The variousdevices described as the sink devices may serve as the source devices.If the sink device has a microphone, a delay period can be measured. Thepresent delay correcting method is also effective when the sink devicedoes not have any speaker which is hardware for outputting sound. Inthis case, the sink device does not generate sound, and the presentmethod is used to synchronize images between the sink and the source.

What is claimed is:
 1. An electronic device comprising: a displayconfigured to display images; an audio output module configured toreproduce an audio signal to output a first audio output; a transmissionmodule configured to transmit the audio signal to a partner device,wherein the partner device is configured to reproduce the transmittedaudio signal to output a second audio output; a first detection moduleconfigured to detect the second audio output from the partner device oranother audio output that is output by the partner device; a seconddetection module configured to detect a parameter corresponding to thepositional relationship between the partner device and the electronicdevice; a third detection module configured to detect a time differencebetween the second audio output and the first audio output; and acontroller configured to control at least one of the timing of thetransmission of the audio signal by the transmission module and thetiming of the output of the first audio output by the audio outputmodule in accordance with the time difference detected by the thirddetection module, wherein the controller is further configured to switchwhether to control the timing in accordance with the positionalrelationship between the electronic device and the partner device. 2.The electronic device of claim 1, wherein the second detection module isconfigured to detect the distance from the partner device in accordancewith the second audio output of the partner device or the another audiooutput of the partner device.
 3. The electronic device of claim 1,wherein if the partner device and the electronic device include a knownpositional relationship, the controller is configured to stopcontrolling the timing so that the audio output module of one of theelectronic device and the partner device outputs sound and so that theoutput of sound from the other audio output module is stopped.
 4. Theelectronic device of claim 1, wherein the second detection modulecomprises more than one of detection module selected from the groupconsisting of a microphone, a camera, radio receptionsensitivity/reception quality, a motion sensor, a global positioningsystem (GPS) signal, and radio charge collection system network basestation information or wireless LAN base station information, and aconversion table is configured to be created in accordance with acquiredvalues of more than one of the detection module at the same time.
 5. Theelectronic device of claim 1, wherein if a radio signal is transmittedto the partner device via a repeater, the second detection module isconfigured to be replaced with a positional relationship detectionmodule, wherein the positional relationship detection module is locatedbetween one of the electronic device and the partner device that greatlychanges in radio signal quality with movement and the repeater of theradio signal.
 6. The electronic device of claim 1, wherein thecontroller is configured to switch the timing of the output of the firstreproduction output by the audio output module depending on whethercontent to be transmitted by radio is protected content.
 7. Anelectronic device comprising: a receiving module configured to receivean audio signal from a partner device; a first detection moduleconfigured to detect a second audio output, wherein the second audiooutput is output by an audio output module or the partner device,wherein the audio output module is configured to output a first audiooutput reproduced from the audio signal received by the receivingmodule; a second detection module configured to detect a parametercorresponding to the positional relationship between the partner deviceand the electronic device; a third detection module configured to detecta time difference between the second audio output of the partner devicedetected by the first detection module and the first audio output thatis output by the first audio output module; and a controller configuredto control at least one of the timing of the transmission of the audiosignal by the partner device and the timing of the output of the firstaudio output by the audio output module in accordance with the timedifference detected by the third detection module, and to switch whetheror not to control the timing in accordance with the positionalrelationship between the electronic device and the partner device. 8.The electronic device of claim 7, wherein the second detection module isconfigured to detect the distance from the partner device in accordancewith the audio signal from the partner device received by the receivingmodule and the second audio output of the partner device detected by thefirst detection module.
 9. The electronic device of claim 7, wherein ifthe partner device and the electronic device include a known positionalrelationship, the controller is configured to not control the timing sothat the audio output module of one of the electronic device and thepartner device outputs sound and so that the output of sound from theother audio output module is stopped.
 10. The electronic device of claim7, wherein a conversion table is configured to be created in accordancewith acquired values of more than one of the detection module of thesecond detection module at the same time.
 11. The electronic device ofclaim 7, wherein the second detection module comprises detection moduleother than the detection of audio information used in a training periodto cerate a conversion table.
 12. The electronic device of claim 7,wherein the second detection module comprises more than one of detectionmodule selected from the group consisting of a microphone, a camera,radio reception sensitivity/reception quality, a motion sensor, a globalpositioning system (GPS) signal, and radio charge collection systemnetwork base station information or wireless LAN base stationinformation, and a conversion table is configured to be created inaccordance with acquired values of more than one of the detection moduleat the same time.
 13. An audio output method of reproducing an audiosignal by a first reproducer and a second reproducer, the firstreproducer outputting a first audio output corresponding to the audiosignal and displaying images corresponding to a video signal, the secondreproducer receiving at least the audio signal and outputting a secondaudio output independently of the first audio reproduction output, theaudio output method comprising: receiving, by the first reproducer, thesecond audio output that is output by the second reproducer; detecting atime difference between the first audio output and the second audiooutput; detecting the distance between the first reproducer and thesecond reproducer in accordance with the detected time difference; andsetting conditions to transmit the audio signal to the second reproducerin accordance with the detected distance between the first reproducerand the second reproducer.
 14. The audio output method of claim 13,wherein if an output module for the audio output of one of the firstreproducer and the second reproducer is a device driven by anoncommercial power source, the audio output is output from the outputmodule for the audio output of the device driven by a commercial powersource.
 15. The audio output method of claim 14, wherein more than onestates are measured as the states of a sound source in accordance withwhether the audio output is output from the first reproducer or thesecond reproducer, and the time auto-correlations are then found,whereby a delay period is measured, and a delay period to be correctedis found.
 16. The audio output method of claim 13, wherein an outputmodule for the audio output of one of the first reproducer or the secondreproducer includes a dual mechanism and circuit configuration, and isswitched to the function of an audio input module on a time axis and isused accordingly.
 17. The audio output method of claim 16, wherein oneof the audio input module of the first reproducer and the audio inputmodule of the second reproducer which is closer to a detection positionof the audio reproduction output is used as the audio input module.